Profiling Data at Table Level

You can assess your data quality by profiling the data at table level. You need to schedule a data profiling job and provide the data quality score by assessing the data quality.

To profile data at table level, follow these steps:

  1. Go to Application Menu > Data Catalog > Metadata Manager.
  2. Under the System Catalogue pane, click a table.
  3. Click Data Quality.

    4. The Data Profiling page appears.

  4. Select columns.
  5. Click the Profile Data button.

    6. The User Credentials page appears. For more information on enforcement of user credentials, refer to the Enforcing Credentials for Data Access or Preview topic.

  6. Enter credentials to connect with the database.

    7. The Job Scheduler page appears.

  7. Enter appropriate values to the fields. Fields marked with red asterisk are mandatory. Refer to the following table for field descriptions.

    8. Option

    Description

    Job Name

    Specifies the job name.

    For example, Administrator1585030550001.

    This field autopopulates with a job name. You can edit it and enter a different job name.

    Interval

    Specifies the frequency of the job.

    Valid values are:

    • Once
    • Every Day
    • Every Week
    • Every Month

    Scheduled Job On

    Set the date and time of the job using .

    For example, 03-24-2020 11:45.

    Local or Server

    Select the machine whose clock decides the time of the scheduled scan.

    • Local: Refers to your local machine.
    • Server: Refers to the machine where erwinDIS has been deployed.

    Data Profile Preferences

    Select the corresponding check boxes to give your data profile preferences in the profile grid report.

    • Total Values: Select the check box to display the total number of rows in the selected columns.
    • Distinct Values: Select the check box to display the number of distinct values in the selected columns.
    • Repeated Values: Select the check box to display the number of repeated values in the selected columns.
    • Null Values: Select the check box to display the number of null values in the selected columns.
    • Minimum Value: Select the check box to display the minimum value in the selected columns. You can enable or disable analysis of minimum value for character data. For more information on this, refer to the Configuring Data Profiling and DQ Scores topic.
    • Maximum Value: Select the check box to display the maximum value in the selected columns. For more information on this, refer to the Configuring Data Profiling and DQ Scores topic.
    • Most Frequent Value: Select the check box to display the most frequent values in the selected columns.
    • Least Frequent Value: Select the check box to display the least frequent values in the selected columns.
    • Most Frequent Patterns: Select the check box to display the most frequent patterns in the selected columns. For more information on this, refer to the Configuring Data Profiling and DQ Scores topic.
    • Least Frequent Patterns: Select the check box to display the least frequent patterns in the selected columns. For more information on this, refer to the Configuring Data Profiling and DQ Scores topic.

     

    Notify Me

    Switch Notify Me to ON to receive email notification.

    For more information on this, refer to the Configuring Notification on Profiling Data topic.

    Notification Email

    This field is autopopulated with your email ID.

    If you enable notifications in the Metadata Manager Settings, you can receive email notifications from the administrator's email ID about the scheduled job.

    CC list

    Enter a comma-separated list of email IDs that should receive email notifications about the scheduled job.

    For example, ab.dav@xyz.com, cal.kai@xyz.com
  8. Click Schedule.

    9. The data profiling job is scheduled.

    The data profiling job is completed at the scheduled time and the job state changes to COMPLETED.

  9. Use the following options:
    Data Profiling Summary Report
    To view data profiling summary, click Data Profiling Summary Report.

    Data Profiling Summary page appears.

    Data Profiling Pattern Summary
    To view data profiling pattern summary report, click Data Profiling Pattern Summary Report.The Data Profiling Pattern Summary page appears.
    Data Profile Statistics
    To view data profile statistics, click Data Profile Statistics.
    The following page appears with data profile statistics.
    Click DQ Score to update data quality score. The Update DQ Score page appears.
    Select DQ Score and click Save. The DQ Score is updated.